101 research outputs found
Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy
Large convolutional neural networks (CNN) can be difficult to train in the
differentially private (DP) regime, since the optimization algorithms require a
computationally expensive operation, known as the per-sample gradient clipping.
We propose an efficient and scalable implementation of this clipping on
convolutional layers, termed as the mixed ghost clipping, that significantly
eases the private training in terms of both time and space complexities,
without affecting the accuracy. The improvement in efficiency is rigorously
studied through the first complexity analysis for the mixed ghost clipping and
existing DP training algorithms.
Extensive experiments on vision classification tasks, with large ResNet, VGG,
and Vision Transformers, demonstrate that DP training with mixed ghost clipping
adds memory overhead and slowdown to the standard
non-private training. Specifically, when training VGG19 on CIFAR10, the mixed
ghost clipping is faster than state-of-the-art Opacus library with
larger maximum batch size. To emphasize the significance of
efficient DP training on convolutional layers, we achieve 96.7\% accuracy on
CIFAR10 and 83.0\% on CIFAR100 at using BEiT, while the previous
best results are 94.8\% and 67.4\%, respectively. We open-source a privacy
engine (\url{https://github.com/JialinMao/private_CNN}) that implements DP
training of CNN with a few lines of code.Comment: Accepted to NeurIPS 202
Towards A Robust Group-level Emotion Recognition via Uncertainty-Aware Learning
Group-level emotion recognition (GER) is an inseparable part of human
behavior analysis, aiming to recognize an overall emotion in a multi-person
scene. However, the existing methods are devoted to combing diverse emotion
cues while ignoring the inherent uncertainties under unconstrained
environments, such as congestion and occlusion occurring within a group.
Additionally, since only group-level labels are available, inconsistent emotion
predictions among individuals in one group can confuse the network. In this
paper, we propose an uncertainty-aware learning (UAL) method to extract more
robust representations for GER. By explicitly modeling the uncertainty of each
individual, we utilize stochastic embedding drawn from a Gaussian distribution
instead of deterministic point embedding. This representation captures the
probabilities of different emotions and generates diverse predictions through
this stochasticity during the inference stage. Furthermore,
uncertainty-sensitive scores are adaptively assigned as the fusion weights of
individuals' face within each group. Moreover, we develop an image enhancement
module to enhance the model's robustness against severe noise. The overall
three-branch model, encompassing face, object, and scene component, is guided
by a proportional-weighted fusion strategy and integrates the proposed
uncertainty-aware method to produce the final group-level output. Experimental
results demonstrate the effectiveness and generalization ability of our method
across three widely used databases.Comment: 11 pages,3 figure
Constrained Reinforcement Learning for Dynamic Material Handling
As one of the core parts of flexible manufacturing systems, material handling
involves storage and transportation of materials between workstations with
automated vehicles. The improvement in material handling can impulse the
overall efficiency of the manufacturing system. However, the occurrence of
dynamic events during the optimisation of task arrangements poses a challenge
that requires adaptability and effectiveness. In this paper, we aim at the
scheduling of automated guided vehicles for dynamic material handling.
Motivated by some real-world scenarios, unknown new tasks and unexpected
vehicle breakdowns are regarded as dynamic events in our problem. We formulate
the problem as a constrained Markov decision process which takes into account
tardiness and available vehicles as cumulative and instantaneous constraints,
respectively. An adaptive constrained reinforcement learning algorithm that
combines Lagrangian relaxation and invalid action masking, named RCPOM, is
proposed to address the problem with two hybrid constraints. Moreover, a
gym-like dynamic material handling simulator, named DMH-GYM, is developed and
equipped with diverse problem instances, which can be used as benchmarks for
dynamic material handling. Experimental results on the problem instances
demonstrate the outstanding performance of our proposed approach compared with
eight state-of-the-art constrained and non-constrained reinforcement learning
algorithms, and widely used dispatching rules for material handling.Comment: accepted by the 2023 International Joint Conference on Neural
Networks (IJCNN
A picture of the space of typical learnable tasks
We develop information geometric techniques to understand the representations
learned by deep networks when they are trained on different tasks using
supervised, meta-, semi-supervised and contrastive learning. We shed light on
the following phenomena that relate to the structure of the space of tasks: (1)
the manifold of probabilistic models trained on different tasks using different
representation learning methods is effectively low-dimensional; (2) supervised
learning on one task results in a surprising amount of progress even on
seemingly dissimilar tasks; progress on other tasks is larger if the training
task has diverse classes; (3) the structure of the space of tasks indicated by
our analysis is consistent with parts of the Wordnet phylogenetic tree; (4)
episodic meta-learning algorithms and supervised learning traverse different
trajectories during training but they fit similar models eventually; (5)
contrastive and semi-supervised learning methods traverse trajectories similar
to those of supervised learning. We use classification tasks constructed from
the CIFAR-10 and Imagenet datasets to study these phenomena
The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
We develop information-geometric techniques to analyze the trajectories of
the predictions of deep networks during training. By examining the underlying
high-dimensional probabilistic models, we reveal that the training process
explores an effectively low-dimensional manifold. Networks with a wide range of
architectures, sizes, trained using different optimization methods,
regularization techniques, data augmentation techniques, and weight
initializations lie on the same manifold in the prediction space. We study the
details of this manifold to find that networks with different architectures
follow distinguishable trajectories but other factors have a minimal influence;
larger networks train along a similar manifold as that of smaller networks,
just faster; and networks initialized at very different parts of the prediction
space converge to the solution along a similar manifold
Recommended from our members
Sources of carbonaceous aerosols and deposited black carbon in the Arctic in winter-spring: implications for radiative forcing
We use a global chemical transport model (GEOS-Chem CTM) to interpret observations of black carbon (BC) and organic aerosol (OA) from the NASA ARCTAS aircraft campaign over the North American Arctic in April 2008, as well as longer-term records in surface air and in snow (2007–2009). BC emission inventories for North America, Europe, and Asia in the model are tested by comparison with surface air observations over these source regions. Russian open fires were the dominant source of OA in the Arctic troposphere during ARCTAS but we find that BC was of prevailingly anthropogenic (fossil fuel and biofuel) origin, particularly in surface air. This source attribution is confirmed by correlation of BC and OA with acetonitrile and sulfate in the model and in the observations. Asian emissions are the main anthropogenic source of BC in the free troposphere but European, Russian and North American sources are also important in surface air. Russian anthropogenic emissions appear to dominate the source of BC in Arctic surface air in winter. Model simulations for 2007–2009 (to account for interannual variability of fires) show much higher BC snow content in the Eurasian than the North American Arctic, consistent with the limited observations. We find that anthropogenic sources contribute 90% of BC deposited to Arctic snow in January-March and 60% in April–May 2007–2009. The mean decrease in Arctic snow albedo from BC deposition is estimated to be 0.6% in spring, resulting in a regional surface radiative forcing consistent with previous estimates.Earth and Planetary SciencesEngineering and Applied Science
Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance
Funding Information: This work was funded by a grant from the National Heart, Lung, and Blood Institute (NHLBI; grant number 1R01HL149948). The funding agency was not involved in the design of the study, collection and analysis of data, interpretation of results, or writing of the manuscript. Publisher Copyright: © 2023, The Author(s).Peer reviewedPublisher PD
- …